Abstract Body - Paper 1 - 

Gathering teacher’s perceptions of evaluator feedback 


Background / Context: 

In response to initiatives to increase educator effectiveness states throughout the nation are placing 
greater emphasis on teacher evaluation tools that differentiate teacher effectiveness and include 
timely and constructive feedback. One of the three principles of the Elementary and Secondary 
Education Act (ESEA) flexibility waivers requires that states develop and implement teacher 
evaluation and support systems that “provide clear, timely, and useful feedback, including 
feedback that identifies needs and guides professional development” (U.S. Department of 
Education, 2012, p. 6). As states and districts begin to develop and implement new teacher 
evaluation systems, they need to identify ways to use evaluation findings to provide individualized 
feedback that will facilitate professional development (Kane & Staiger, 2012). 

In response to these needs REL Central developed a research project in collaboration with 
members of the educator effectiveness research alliance to examine perceptions of evaluator 
feedback. This presentation will describe efforts to develop and test a survey to gather teacher’s 
perceptions of the feedback they receive from their evaluators, specifically their perceptions 
usefulness, accuracy, and credibility of the feedback. The presentation will provide infonnation on 
the iterative process used to develop and test the survey and will also describe how the survey is 
being used in research and evaluation efforts. 

Purpose / Objective / Research Question / Focus of Study: 

REL Central conducted a study during the 2014/15 school year to examine relationships among 
feedback characteristics (perceived utility and accuracy of evaluator feedback and credibility of the 
person providing the feedback), access to resources related to the feedback, teacher response to 
feedback, and teacher performance. The Examining Evaluator Feedback Survey was developed to 
collect infonnation for that study. The theoretical framework used to develop the survey is 
provided in appendix B (figure Bl). A list of survey questions is provided in appendix B (Table 
Bl) 

Setting: 

The survey was pilot tested in school districts in two states in the central region who were 
implementing new teacher evaluation systems focused on providing timely and relevant feedback 
to teachers in order to increase perfonnance. 

Population / Participants / Subjects: 

The subjects included an advisory panel, cognitive interview participants and survey respondents. 
The advisory panel comprised seven members, including expert survey developers, state leaders, 
and district leaders with oversight for educator evaluation systems. Nine teachers were interviewed 
by the study team using a structured interview protocol. Following the cognitive interviews, the 
revised survey was administered to 196 teachers, 190 of whom completed the full survey. 
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Research Design: 


The survey was developed using an iterative process that included such survey development 
methods as expert review, cognitive interviews, and statistical modeling (Presser et ah, 2004; 
Rothgeb, 2008). The original survey questions were formulated based on previous research as well 
as unpublished surveys that the study team had implemented in evaluations of various teacher 
evaluation systems. The questions were reviewed and revised based on feedback from an advisory 
committee and teachers. The survey was then administered to 196 teachers, and the results were 
used to examine its reliability and validity. The stages of development are described in Appendix 
B (figure B2) 


Data Collection and Analysis: 

Researchers conducted an expert panel review and cognitive interviews to examine the clarity of 
the survey questions to ensure that the survey items were understood and easy to answer. To 
detennine whether the survey was relevant to teacher evaluation systems in different contexts and 
whether the survey had face validity, the study team conducted a webinar with an advisory panel in 
which the panel responded to questions about the clarity and applicability of the questions and the 
appropriateness of the directions and response options. To determine whether the survey questions 
were clear and would be uniformly interpreted by teachers, the study team conducted cognitive 
interviews with a sample of teachers. Following the cognitive interviews, the revised survey was 
administered to teachers. These data were used to examine the reliability and validity of the 
survey. Survey reliability and validity were examined using classical test theory, Rasch analysis, 
and confirmatory factor analysis. Analyses were conducted on the questions related to usefulness, 
accuracy, credibility, access to resources, and responsiveness. 


Findings / Results: 

The analyses provided evidence for the reliability and validity of the questions and the categories 
of Usefulness, Accuracy, Credibility, Access to Resources, and Responsiveness to infonn state and 
district leaders about teachers’ perceptions of evaluator feedback. These categories showed high 
internal consistency, with Cronbach’s alphas of 0.827-0.939. Confirmatory factor analysis 
suggested that the questions in these categories represent five distinct, though interrelated, 
categories. The importance of feedback questions are numbered in question sets because they share 
the same question prompts. The prompts were used to make the survey easier to take. The 
importance of feedback questions showed high internal consistency, with a Cronbach’s alpha of 
0.931. 
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The final survey is scheduled to be released in the fall of 20 1 5 as a tool with guidance on how 
states and districts can use the survey gather data on teacher’s perceptions of evaluator feedback. 
The survey was administered as part of a REL Central research study for which the data is 
currently being analyzed to examine the relationship among the characteristics of perfonnance 
feedback (usefulness, accuracy, and credibility) and teacher’s responses to feedback. The survey 
was also administered by the Missouri Department of Elementary and Secondary Education to 
evaluators in Missouri prior to and after they attended professional development on providing 
feedback. 


Conclusions: 

The examining evaluator feedback survey is designed to gather teacher’s perceptions of the 
feedback they receive from their evaluators. The survey is being used to study theories of 
perfonnance feedback that have limited research in the education field to provide states and 
districts with useful information about teacher’s perceptions of evaluator feedback and factors that 
are related to teacher’s use of feedback. The survey is also a tool that states and districts can use to 
gather perceptions of feedback in their own settings. 
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Appendix B. Tables and Figures 


Inputs 


• Feedback characteristics: 

•Teachers receive feedback 
that they perceive as useful. 

•Teachers receive feedback 
that they believe accurately 
represents their 
performance. 

•Teachers believe that the 
person providing feedback 
is qualified to do so. 


Intermediate Outcomes 
(Mediating Variables) 


•Teachers have access to the 
necessary opportunities to 
learn about or plan for 
implementation of feedback. 

•Teachers implement changes 
or seek professional 
development based on 
feedback. 


Final Outcomes 


•Supervisors/evaluators 
rate teachers' 
performance as proficient, 
distinguished, or 
accomplished. 


Figure Bl. Theoretical Framework for Examining Evaluator Feedback Study 
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Table Bl. List of Examining Evaluator Feedback Survey Questions 




Question 


Section 

Description 

number 



Definition of designated evaluator 

1 


Background information 

Designated teacher evaluator in the current school year 

2 



Frequency of feedback conversation with designated evaluator 

3 



Frequency of written feedback from designated evaluator 

4 


Feedback 

characteristics (Includes 

Usefulness: perceived usefulness of evaluator’s feedback 

5 (a-g) 


Accuracy: perceived accuracy of evaluator’s feedback 

6 (a-d) 


five categories of 

questions) 

Credibility: perceived credibility of evaluator 

7 (a-e) 



Access to resources: perceived access to professional development and other resources 
needed to respond to evaluation feedback 

8 (a-d) 



Responsiveness: actions teacher took in response to evaluation feedback 

9 (a-e) 


Importance of feedback 

Importance of the following characteristics when deciding how to respond to the 

10 (a-i) 


characteristics 

feedback: 

11 (a-b) 



• Perceiving the feedback as useful. 

12 (a-e) 



• Having confidence in the accuracy of the evaluation feedback. 

• Perceiving the evaluator as credible. 

• Having access to relevant resources. 

13 (a-d) 


Belief about 
instructional 
improvement 

Belief about whether evaluation feedback improved teacher’s instruction 

14 


Teacher demographics 

Number of years teaching 

15 



Grade level or levels currently teaching 

16 



Subject area or areas currently teaching 

17 
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Figure B2. Survey Development Process 
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